Event record tracking across multiple search sessions

ABSTRACT

A method, system, and processor-readable storage medium are directed towards generating a report derived from data, such as event data, stored on a plurality of distributed nodes. In one embodiment the analysis is generated using a “divide and conquer” algorithm, such that each distributed node analyzes locally stored event data while an aggregating node combines these analysis results to generate the report. In one embodiment, each distributed node also transmits a list of event data references associated with the analysis result to the aggregating node. The aggregating node may then generate a global ordered list of data references based on the list of event data references received from each distributed node. Subsequently, in response to a user selection of a range of global event data, the report may dynamically retrieve event data from one or more distributed nodes for display according to the global order.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims benefit as a Continuation of application Ser.No. 13/660,845, filed on Oct. 25, 2012, which claims benefit as aContinuation of application Ser. No. 13/223,167, filed on Aug. 31, 2011,which claims benefit of Provisional application Ser. No. 61/452,591,filed on Mar. 14, 2011, the entire contents of the aforementioned arehereby incorporated by reference as if fully set forth herein, under 35U.S.C. §120. The applicant(s) hereby rescind any disclaimer of claimscope in the parent application(s) or the prosecution history thereofand advise the USPTO that the claims in this application may be broaderthan any claim in the parent application(s).

TECHNICAL FIELD

The present invention relates generally to generating reports and moreparticularly but not exclusively to reducing latency when generatinginteractive reports from data contained on a plurality of distributedcomputing nodes.

BACKGROUND

An increasing number of computing applications, particularly within theenterprise, entail analyzing distributed data. One type of analysis isreport generation, such as generating a table, a chart, or a timelinefrom distributed data. Some reports are generated by copying all of thedata from the plurality of distributed nodes to a single aggregatingnode for analysis. However, if the amount of data to be analyzed islarge, then transmitting this data over a network can be prohibitivelytime consuming. Also, a single aggregating node may not contain thecapacity necessary to store data received from a plurality ofdistributed nodes. Accordingly, efficiently generating reports thatcontain aggregate information as well as raw data is an ongoingchallenge.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments are described with referenceto the following drawings. In the drawings, like reference numeralsrefer to like parts throughout the various figures unless otherwisespecified.

For a better understanding of the described embodiments, reference willbe made to the following Detailed Description, which is to be read inassociation with the accompanying drawings, wherein:

FIG. 1 illustrates a system diagram of an environment in whichembodiments of the invention may be implemented;

FIG. 2 illustrates an embodiment of a client device that may be includedin a system such as that shown in FIG. 1;

FIG. 3 illustrates an embodiment of a network device that may beincluded in a system such as that shown in FIG. 1;

FIG. 4 illustrates a logical flow diagram showing one embodiment of aprocess for generating and displaying an interactive report;

FIG. 5 illustrates a logical flow diagram showing one embodiment of aprocess for analyzing raw data on a distributed node for display in aninteractive report; and

FIG. 6 illustrates one non-limiting embodiment of an interactive report.

DETAILED DESCRIPTION OF THE INVENTION

Throughout the specification and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise. The phrase “in one embodiment” as used herein doesnot necessarily refer to the same embodiment, though it may.Furthermore, the phrase “in another embodiment” as used herein does notnecessarily refer to a different embodiment, although it may. Thus, asdescribed below, various embodiments of the invention may be readilycombined, without departing from the scope or spirit of the invention.

In addition, as used herein, the term “or” is an inclusive “or”operator, and is equivalent to the term “and/or,” unless the contextclearly dictates otherwise. The term “based on” is not exclusive andallows for being based on additional factors not described, unless thecontext clearly dictates otherwise. In addition, throughout thespecification, the meaning of “a,” “an,” and “the” include pluralreferences. The meaning of “in” includes “in” and “on.”

As used herein, the term “event data” refers to computing data that iscollected about a computing system, including, for example, an action,characteristic, condition (or state), or state change of the computingsystem. For example, such events may be about a computing system'sperformance, actions taken by the computing system, or the like. Eventdata may be obtained from various computing log files generated by thecomputer's operating system, and/or other monitoring application.However, event data is not restricted by a file format or structure fromwhich the event data is obtained.

As used herein, an event record refers to data associated with a singleevent.

As used herein, the term “report” refers to one or more visualizationsof search query results. For example, a report may include a table ofdata, a timeline, a chart, a “field picker” or the like. In oneembodiment, the report is interactive, enabling a user to selectivelyview pieces of raw data used to generate the report. For example, if thereport lists users sorted based on the number of times each user haslogged into the system, each user is selectable to view detailed recordsof that user's login events.

Briefly described is a mechanism for generating a report derived fromdata, such as event data, stored on a plurality of distributed nodes. Inone embodiment the analysis is generated using a “divide and conquer”algorithm, such that each distributed node analyzes locally stored eventdata while an aggregating node combines these analysis results togenerate the report. In one embodiment, each distributed node alsotransmits a list of event data references associated with the analysisresult to the aggregating node. The aggregating node may then generate aglobal ordered list of data references based on the list of event datareferences received from each distributed node. Subsequently, inresponse to a user selection of a range of global event data, the reportmay dynamically retrieve event data from one or more distributed nodesfor display according to the global order.

Illustrative Operating Environment

FIG. 1 shows components of one embodiment of an environment in which theinvention may be practiced. Not all the components may be required topractice the invention, and variations in the arrangement and type ofthe components may be made without departing from the spirit or scope ofthe invention. As shown, system 100 of FIG. 1 includes local areanetworks (“LANs”)/wide area networks (“WANs”)-(network) 107, clientdevices 101-103, and distributed search server 109.

One embodiment of client devices 101-103 is described in more detailbelow in conjunction with FIG. 2. Generally, however, client devices101-103 may include virtually any computing device capable ofcommunicating over a network to send and receive information, includinga search query, analysis results of a search query, lists of event datareferences, collections of event data, and the like. Client devices101-103 are referred to interchangeably herein as “distributed computingdevices”, “distributed nodes”, or the like. In one embodiment, one ormore of client devices 101-103 may be configured to operate within abusiness or other entity to perform a variety of services for thebusiness or other entity. For example, client devices 101-103 may beconfigured to operate as a web server, an accounting server, aproduction server, an inventory server, or the like. However, clientdevices 101-103 are not constrained to these services and may also beemployed, for example, as an end-user computing node, in otherembodiments. Further, it should be recognized that more or less clientdevices may be included within a system such as described herein, andembodiments are therefore not constrained by the number or type ofclient devices employed.

The set of such client devices 101-103 may include devices thattypically connect using a wired or wireless communications medium suchas personal computers, servers, multiprocessor systems,microprocessor-based or programmable consumer electronics, network PCs,or the like. In one embodiment, at least some of client devices 101-103may operate over wired and/or wireless network. In some embodiments,client devices 101-103 may include virtually any portable computingdevice capable of receiving and sending a message over a network, suchas network 107.

Client devices 101-103 also may include at least one client applicationthat is configured to capture and record event data and/or relatedmetadata. However, the client application need not be limited to merelyproviding event data and related metadata, and may also provide otherinformation, and/or provide for a variety of other services, including,for example, monitoring for events within and/or between client devices.

The client application may further provide information that identifiesitself, including a type, capability, name, and the like. Suchinformation may be provided in a network packet, or the like, sentbetween other client devices, distributed search server 109, or othercomputing devices.

Network 107 is configured to couple network devices with other computingdevices, including distributed search server 109 and client devices101-103. Network 107 is enabled to employ any form of computer readablemedia for communicating information from one electronic device toanother. Also, network 107 can include the Internet in addition to localarea networks (LANs), wide area networks (WANs), direct connections,such as through a universal serial bus (USB) port, other forms ofcomputer-readable media, or any combination thereof. On aninterconnected set of LANs, including those based on differingarchitectures and protocols, a router acts as a link between LANs,enabling messages to be sent from one to another. In addition,communication links within LANs typically include twisted wire pair orcoaxial cable, while communication links between networks may utilizeanalog telephone lines, full or fractional dedicated digital linesincluding T1, T2, T3, and T4, and/or other carrier mechanisms including,for example, E-carriers, Integrated Services Digital Networks (ISDNs),Digital Subscriber Lines (DSLs), wireless links including satellitelinks, or other communications links known to those skilled in the art.Moreover, communication links may further employ any of a variety ofdigital signaling technologies, including without limit, for example,Digital Signal (DS)-0, DS-1, DS-2, DS-3, DS-4, Optical Carrier (OC)-3,OC-12, OC-48, or the like. Furthermore, remote computers and otherrelated electronic devices could be remotely connected to either LANs orWANs via a modem and temporary telephone link. In one embodiment,network 107 may be configured to transport information of an InternetProtocol (IP). In essence, network 107 includes any communication methodby which information may travel between computing devices.

Additionally, communication media typically embodies computer-readableinstructions, data structures, program modules, or other transportmechanism and includes any information delivery media. By way ofexample, communication media includes wired media such as twisted pair,coaxial cable, fiber optics, wave guides, and other wired media andwireless media such as acoustic, Radio Frequency (RF), infrared, andother wireless media.

In some embodiments, network 107 may be further configurable as awireless network, which may further employ a plurality of accesstechnologies including 2nd (2G), 3rd (3G), 4th (4G) generation radioaccess for cellular systems, WLAN, Wireless Router (WR) mesh, and thelike. In one non-limiting example, network 107, when configured as awireless network, may enable a radio connection through a radio networkaccess such as Global System for Mobile communication (GSM), GeneralPacket Radio Services (GPRS), Enhanced Data GSM Environment (EDGE),Wideband Code Division Multiple Access (WCDMA), and the like.

Distributed search server 109 includes virtually any network deviceusable to receive a search query, distribute sub-queries of the searchquery among client devices 101-103, synthesize the results of thesub-queries, and display a report. Distributed search server 109 may,for example, be configured to merge lists of event data references intoa global ordered list of event data references, enabling ranges of eventdata to be selectively retrieved from one or more distributed nodes.

Devices that may operate as distributed search server 109 includevarious network devices, including, but not limited to personalcomputers, desktop computers, multiprocessor systems,microprocessor-based or programmable consumer electronics, network PCs,server devices, network appliances, and the like.

Although FIG. 1 illustrates distributed search server 109 as a singlecomputing device, the invention is not so limited. For example, one ormore functions of the distributed search server 109 may be distributedacross one or more distinct network devices. Moreover, distributedsearch server 109 is not limited to a particular configuration. Thus, inone embodiment, distributed search server 109 may contain a plurality ofnetwork devices to perform digest aggregation and calculation ofapproximate order statistics therefrom. Similarly, in anotherembodiment, distributed search server 109 may operate as a plurality ofnetwork devices within a cluster architecture, a peer-to-peerarchitecture, and/or even within a cloud architecture. Thus, theinvention is not to be construed as being limited to a singleenvironment, and other configurations, and architectures are alsoenvisaged.

Illustrative Client Device

FIG. 2 shows one embodiment of client device 200 that may be included ina system implementing embodiments of the invention. Client device 200may include many more or less components than those shown in FIG. 2.However, the components shown are sufficient to disclose an illustrativeembodiment for practicing the present invention. Client device 200 mayrepresent, for example, one embodiment of at least one of client devices101-103 of FIG. 1.

As shown in the figure, client device 200 includes processing unit (CPU)202 in communication with a mass memory 226 via a bus 234. Client device200 also includes a power supply 228, one or more network interfaces236, an audio interface 238, a display 240, and an input/outputinterface 248. Power supply 228 provides power to client device 200.

Network interface 236 includes circuitry for coupling client device 200to one or more networks, and is constructed for use with one or morecommunication protocols and technologies including, but not limited to,global system for mobile communication (GSM), code division multipleaccess (CDMA), time division multiple access (TDMA), user datagramprotocol (UDP), transmission control protocol/Internet protocol(TCP/IP), Short Message Service (SMS), general packet radio service(GPRS), Wireless Application Protocol (WAP), ultra wide band (UWB),Institute of Electrical and Electronics Engineers (IEEE) 802.16Worldwide Interoperability for Microwave Access (WiMax), SessionInitiation Protocol (SIP)/Real-time Transport Protocol (RTP), or any ofa variety of other communication protocols. Network interface 236 issometimes known as a transceiver, transceiving device, or networkinterface card (NIC).

Audio interface 238 is arranged to produce and receive audio signalssuch as the sound of a human voice. For example, audio interface 238 maybe coupled to a speaker and microphone (not shown) to enabletelecommunication with others and/or generate an audio acknowledgementfor some action. Display 240 may be a liquid crystal display (LCD), gasplasma, light emitting diode (LED), or any other type of display usedwith a computing device. Display 240 may also include a touch sensitivescreen arranged to receive input from an object such as a stylus or adigit from a human hand.

Client device 200 also comprises input/output interface 248 forcommunicating with external devices, such as a keyboard, or other inputor output devices not shown in FIG. 2. Input/output interface 248 canutilize one or more communication technologies, such as USB, infrared,Bluetooth™, or the like.

Mass memory 226 includes a Random Access Memory (RAM) 204, a Read OnlyMemory (ROM) 222, and other storage means. Mass memory 226 illustratesan example of computer readable storage media (devices) for storage ofinformation such as computer readable instructions, data structures,program modules or other data. Mass memory 226 stores a basicinput/output system (“BIOS”) 224 for controlling low-level operation ofclient device 200. The mass memory also stores an operating system 206for controlling the operation of client device 200. It will beappreciated that this component may include a general-purpose operatingsystem such as a version of UNIX, or LINUX™, or a specialized clientcommunication operating system such as Windows Mobile™, or the Symbian®operating system. The operating system may include, or interface with aJava virtual machine module that enables control of hardware componentsand/or operating system operations via Java application programs.

Mass memory 226 further includes one or more data storage 208, which canbe utilized by client device 200 to store, among other things,applications 214 and/or other data. For example, data storage 208 mayalso be employed to store information that describes variouscapabilities of client device 200. The information may then be providedto another device based on any of a variety of events, including beingsent as part of a header during a communication, sent upon request, orthe like. At least a portion of the information may also be stored on adisk drive or other computer-readable storage device 230 within clientdevice 200. Data storage 208 may further store event data and metadata210 and local search results 212. Such event data and metadata 210 andlocal search results 212 may also be stored within any of a variety ofother computer-readable storage devices, including, but not limited to ahard drive, a portable storage device, or the like, such as illustratedby computer-readable storage device 230.

Applications 214 may include computer executable instructions which,when executed by client device 200, transmit, receive, and/or otherwiseprocess network data. Other examples of application programs includecalendars, search programs, email clients, IM applications, SMSapplications, Voice Over IP (VOIP) applications, contact managers, taskmanagers, transcoders, database programs, word processing programs,security applications, spreadsheet programs, games, search programs,data log recording programs, and so forth. Applications 214 may include,for example, local search module 220. Local search module 220 mayprocess a sub-query, returning analysis results and a list of event datareferences associated with the analysis results, as described herein.

Illustrative Network Device

FIG. 3 shows one embodiment of a network device 300, according to oneembodiment of the invention. Network device 300 may include many more orless components than those shown. The components shown, however, aresufficient to disclose an illustrative embodiment for practicing theinvention. Network device 300 may be configured to operate as a server,client, peer, or any other device. Network device 300 may represent, forexample distributed search server 109 of FIG. 1.

Network device 300 includes processing unit 302, an input/outputinterface 332, video display adapter 336, and a mass memory, all incommunication with each other via bus 326. The mass memory generallyincludes RAM 304, ROM 322 and one or more permanent mass storagedevices, such as hard disk drive 334, tape drive, optical drive, and/orfloppy disk drive. The mass memory stores operating system 306 forcontrolling the operation of network device 300. Any general-purposeoperating system may be employed. Basic input/output system (“BIOS”) 324is also provided for controlling the low-level operation of networkdevice 300. As illustrated in

FIG. 3, network device 300 also can communicate with the Internet, orsome other communications network, via network interface unit 330, whichis constructed for use with various communication protocols includingthe Transmission Control Protocol/Internet Protocol (TCP/IP) protocol.Network interface unit 330 is sometimes known as a transceiver,transceiving device, or network interface card (NIC).

Network device 300 also comprises input/output interface 332 forcommunicating with external devices, such as a keyboard, or other inputor output devices not shown in FIG. 3. Input/output interface 332 canutilize one or more communication technologies, such as USB, infrared,Bluetooth™, or the like.

The mass memory as described above illustrates another type ofcomputer-readable media, namely computer-readable storage media and/orprocessor-readable storage medium. Computer-readable storage media(devices) may include volatile, nonvolatile, removable, andnon-removable media implemented in any method or technology for storageof information, such as computer readable instructions, data structures,program modules, or other data. Examples of computer readable storagemedia include RAM, ROM, EEPROM, flash memory or other memory technology,Compact Disc ROM (CD-ROM), digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other non-transitoryphysical medium which can be used to store the desired information andwhich can be accessed by a computing device.

As shown, data storage 308 may include a database, text, spreadsheet,folder, file, or the like, that may be configured to maintain and storeuser account identifiers, user profiles, email addresses, IM addresses,and/or other network addresses; or the like. Data stores 308 may furtherinclude program code, data, algorithms, and the like, for use by aprocessor, such as central processing unit (CPU) 302 to execute andperform actions. In one embodiment, at least some of data store 308might also be stored on another component of network device 300,including, but not limited to computer-readable storage medium 328, harddisk drive 334, or the like. Data storage 308 may further store orderedlist of event data references 310.

Ordered list of event data references 310 may include a list of eventdata references received from a plurality of distributed nodes. In oneembodiment, the ordered list of event data references is generated bysorting data references received from each distributed node according toa common field, such as a timestamp, a number a string, or the like. Inone embodiment, each element of the ordered list includes a reference tothe distributed node the event data is stored on, an offset or otherpointer to the event data on that distributed node, and optionally thevalue used to sort the ordered list.

The mass memory also stores program code and data. One or moreapplications 314 are loaded into mass memory and run on operating system306. Examples of application programs may include transcoders,schedulers, calendars, database programs, word processing programs,Hyper Text Transfer Protocol (HTTP) programs, customizable userinterface programs, Internet Protocol Security (IPSec) applications,encryption programs, security programs, SMS message servers, accountmanagers, and so forth. Distributed search module 318 may also beincluded as application programs within applications 314.

Distributed search module 318 may be configured and arranged to receivea query, generate sub-queries for each of a specified set of distributeddevices, and aggregate results of these sub-queries to generate areport, as described further herein.

Generalized Operation

The operation of certain aspects will now be described with respect toFIGS. 4-6. FIGS. 4-5 provide logical flow diagrams illustrating certainaspects, while FIG. 6 illustrates an example of a scalable interactivedisplay of distributed data. FIG. 4 illustrates a logical flow diagramof one embodiment of a process for generating and displaying aninteractive report. In one embodiment, process 400 may be implemented ondistributed search server 109.

Process 400 begins, after a start block, at block 402, where a searchquery (hereinafter “query”) is received. In one embodiment, the receivedquery targets data, such as “event data” (also referred to as “events”),that is distributed across a plurality of specified computing devices,such as client devices 101-103. In one embodiment, sub-queries aregenerated for each of the specified computing devices and submitted toeach corresponding computing device for processing. For example, if thereceived query asks for a count of system log entries that contain theword “error”, then a sub-query is generated for each of the specifiedcomputing devices, where each sub-query counts the number of eventsderived from system log entries that contain the word “error” stored onthat device.

The received query may specify which computing devices to search in anumber of ways. In one embodiment, the received query specifiesparticular computing devices or groups of computing devices by name,network address, or the like. In another embodiment, computing devicesare specified based on attributes, such as operating system, hardwarecomponents (e.g. CPU, web cam, network adapter, etc.), form factor (e.g.laptop, desktop, server, tablet, virtual machine, smartphone, etc.), andthe like. In another embodiment, a query may specify all of theplurality of computing devices.

In one embodiment, the received query is received from a user, such as asystem administrator. However, queries may also be automaticallygenerated by a software agent. In one embodiment, a query may beautomatically generated at periodic intervals, such as every hour orevery Saturday night. In another embodiment, a query may be generated inresponse to an event, such as installation of a software patch, or inresponse to a metric crossing a threshold, such as an unusually largevolume of network traffic.

The distributed data to be searched may be stored on the specifiedcomputing devices in many ways. In one embodiment, the distributed datamay include events, as defined herein, that have been recorded andstored by each of the specified computing devices. However, thedistributed data may be generated at any time and in any manner,including partitioning a data set across specified computing devicesafter the query has been received. Also, while in one embodiment thedistributed data comprises “events” or “event data” as defined herein,the distributed data may include any kind of data, structured orunstructured.

The received query may include one or more analyses to be performed onthe distributed event data by the computing devices storing that data.For example, an analysis may include counting the number of events thatsatisfy a condition, deriving statistical information about events(including distributions, histograms, N^(th) percentile rankings, andthe like), grouping events, sorting events, and the like. That is, theanalysis may be performed in response to the query.

In one embodiment, the received query may also specify the order ofquery results. For example, a query requesting system log entries thatcontain the word “error” may be ordered based on the time the system logentry was generated (timestamp). A similar query may order entries basedon an error severity value field in the event derived from the systemlog entry. Multiple orderings and nested orderings are alsocontemplated, such as ordering first by an error severity value and thenby a timestamp.

The process proceeds to block 404, where sub-query results are receivedfrom each of the specified computing devices. In one embodiment, thesub-query results include analysis results corresponding to the one ormore analyses specified in the received query. In one embodiment,analysis results are derived from raw event data stored on each of thespecified devices, but analysis results do not include the actual rawevent data.

The sub-query results additionally include one or more lists of eventreferences. In one embodiment, each event reference includes (1) anidentifier that uniquely identifies the event on the computing devicethat generated it, and (2) a value usable to order the event(hereinafter “order value”). In one embodiment the unique identifierincludes a serial number assigned to the event as the event is created.In this example, the unique identifier is unique to a given computingdevice—events from different computing devices may be assigned the sameunique identifier. However, globally unique identifiers, such as GUIDs,are similarly contemplated.

In one embodiment, the order value of an event may be a timestamp, suchas the time when an event was created. However, any value of any datatype is similarly contemplated. Other examples of order values includeintegers, such as a severity of error, and a string, such as a username.

In one embodiment, a computing device creates an event reference foreach event used to generate an analysis result. For example, considerthree devices A, B, and C that contain 14, 37, and 94 system log eventscontaining the word, “error”, respectively. If a query to count all ofthe system log events that contain the word “error” is received, deviceA will return a count of 14 as well as a list of 14 references, onereference for each of the 14 counted events. Similarly, device B willreturn a count of 37 as well as a list of 37 references, and device Cwill return a count of 94 and a list of 94 references. Note that at thistime, none of the raw event data has been transmitted to the distributedsearch server.

The process proceeds to block 406, where a global ordered list of eventreferences is generated based on each of the returned lists of eventreferences. In one embodiment, each entry in the global ordered listincludes the content of an event reference, as described above, as wellas an identifier of the computing device that the event was found on.

Continuing the example above, consider if the first 7 of device A'sevents were the first to be recorded, followed by the first 50 of deviceC's, followed by all 37 of device B's, followed by the last 44 of deviceC, and finally the last 7 of device A. In this simple example, theglobal ordered list would include all 145 event references in this sameorder, where each event reference is fully qualified to include acomputing device identifier in addition to that event's uniqueidentifier. In this way, a user may select a range from the globalordered list of event references, and only the actual event datacontained in the selected range is downloaded.

The process proceeds to block 408, where a request to display a range ofevents is received. Continuing the above example, the global orderedlist includes 145 fully qualified event references. A request may bereceived to display the last 5 events, the second 50 events, the firstevent, all 145 of the events, or any other sub-range of the events.

The process proceeds to block 410, where event data is requested fromone or more of the computing devices based on the range of eventreferences requested from the global ordered list. For example, if thefirst 50 events are requested, then the first 50 entries in the globalordered list are retrieved. Continuing the example above, the first 7events from device A would be requested, all 37 of the events fromdevice B would be requested, and the first 6 events from device C wouldbe requested. Thus a total of 50 events are retrieved from threedifferent computing devices, without retrieving any unnecessary events.In one embodiment these requests are made in parallel, however requestsmay be submitted to individual devices serially. Also, in oneembodiment, a range of events may be requested from a single computingdevice in a single network transaction, however requests may also bemade individually.

The process proceeds to block 412, where the raw data is displayed. Inone embodiment, event data retrieved from individual computing devicesare displayed according to the global order. In one embodiment, therequested raw data is displayed with the one or more analysis results.In this way, a user may see the analysis results as well as portions ofthe underlying data. The process then proceeds to a return block.

FIG. 5 illustrates a logical flow diagram generally showing oneembodiment of a process an individual computing device may perform inthe course of performing a distributed search query. In one embodiment,process 500 is performed by one of client devices 101-103.

Process 500 begins, after a start block, at block 502, where a sub-queryis received from a distributed search server. The process then proceedsto block 504, where data such as events are analyzed according to thereceived sub-query. In one embodiment, as events are analyzed, eventsthat contribute to the requested analysis are referenced in a list ofevent references.

The process then proceeds to block 506, where the results of theanalysis and the list of event references are transmitted to thecomputing device that submitted the sub-query. In one embodiment, thisdevice is distributed search server 109.

The process then proceeds to block 508, where a request for one or morepieces of event data is received. In one embodiment, the requestincludes a contiguous range of event data. In another embodiment,individual pieces of event data are individually requested. The processthen proceeds to block 510, where the requested pieces of event data aretransmitted to the requestor. The process then proceeds to a returnblock.

FIG. 6 illustrates one non-limiting example of an interactive report600; however, other layouts containing other types of information aresimilarly contemplated. The interactive report was generated based on asearch query 602 of all domain name system (dns) lookups the specifiedclients performed “yesterday”. The report is broken into threesections—a timeline 604, a field picker 606, and an event data view 608.The timeline includes a bar graph 610 depicting how many dns lookupswere performed each hour. Field picker 606 is generally used to selectfields 612 from all of the fields available on a given type of event. Inthis example, field picker 606 has been used to select two of the 24fields associated with dns lookup events: client host and client IP.Thus, the event data displayed in the event data view will contain onlythese two fields. Finally, the event data view 608 displays raw eventdata, currently 50 results per page. A total of 562 events were gatheredfrom 79 clients. However, only the first 50 events have been downloadedto the distributed search server at the time this display was generated.If the user were to select another range of 50 events, the distributedsearch server could retrieve these 50 events from one or more of theclients in real-time as discussed above in conjunction with FIGS. 4 and5.

It will be understood that figures, and combinations of steps in theflowchart-like illustrations, can be implemented by computer programinstructions. These program instructions may be provided to a processorto produce a machine, such that the instructions, which execute on theprocessor, create means for implementing the actions specified in theflowchart block or blocks. The computer program instructions may beexecuted by a processor to cause a series of operational steps to beperformed by the processor to produce a computer implemented processsuch that the instructions, which execute on the processor to providesteps for implementing the actions specified in the flowchart block orblocks. These program instructions may be stored on a computer readablemedium or machine readable medium, such as a computer readable storagemedium.

Accordingly, the illustrations support combinations of means forperforming the specified actions, combinations of steps for performingthe specified actions and program instruction means for performing thespecified actions. It will also be understood that each block of theflowchart illustration, and combinations of blocks in the flowchartillustration, can be implemented by modules such as special purposehardware-based systems which perform the specified actions or steps, orcombinations of special purpose hardware and computer instructions.

The above specification, examples, and data provide a completedescription of the manufacture and use of the composition of thedescribed embodiments. Since many embodiments can be made withoutdeparting from the spirit and scope of this description, the embodimentsreside in the claims hereinafter appended.

The invention claimed is:
 1. A method, comprising: receiving, at a firstdevice, a search query to be performed on a set of event recordsaccessible by the first device; searching, by the first device, the setof event records using the search query; sending, by the first device, asearch result to a second device, the search result including one ormore event identifiers, wherein each event identifier of the one or moreevent identifiers is associated with a specific event record accessibleby the first device of the set of event records that satisfied thesearch query, wherein each event identifier enables locating anassociated specific event record accessible by the first device withoutsearching the set of event records; after sending the search result tothe second device, receiving, by the first device from the seconddevice, at least one event identifier that was included in the searchresult, the at least one event identifier sent by the second device inresponse to a user request to view underlying data related to the searchresult; based on receiving the at least one event identifier from thesecond device, sending, by the first device, at least one event recordaccessible by the first device associated with the received at least oneevent identifier to the second device, the at least one event record iscomprised of raw data that relates to operations or activities in aninformation technology environment.
 2. The method as recited in claim 1,wherein each event record in the set of event records is associated witha time stamp.
 3. The method as recited in claim 1, wherein each eventrecord in the set of event records is associated with a time stamp andis searchable based on a time represented by the time stamp.
 4. Themethod as recited in claim 1, wherein an event identifier includes anidentification of both an event record and a device having access to theevent record.
 5. The method as recited in claim 1, wherein the receivedat least one event identifier received by the first device includes anidentification of both an event record and the first device.
 6. Themethod as recited in claim 1, wherein the search result sent by thefirst device to the second device does not include an event recordassociated with an event identifier that was included in the searchresult.
 7. The method as recited in claim 1, wherein the first deviceincludes an indexer.
 8. The method as recited in claim 1, wherein thesecond device includes a search head.
 9. The method as recited in claim1, wherein the first device includes an indexer, and wherein the seconddevice includes a search head.
 10. An apparatus, comprising: a searchquery receiver, at a first device, implemented at least partially inhardware, that receives a search query to be performed on a set of eventrecords accessible by the first device; a subsystem, at the firstdevice, implemented at least partially in hardware, that searches theset of event records using the search query; a subsystem, at the firstdevice, implemented at least partially in hardware, that sends a searchresult to a second device, the search result including one or more eventidentifiers, wherein each event identifier of the one or more eventidentifiers is associated with a specific event record accessible by thefirst device of the set of event records that satisfied the searchquery, wherein each event identifier enables locating an associatedspecific event record accessible by the first device without searchingthe set of event records; a subsystem, at the first device, implementedat least partially in hardware, that after sending the search result tothe second device, receives from the second device, at least one eventidentifier that was included in the search result, the at least oneevent identifier sent by the second device in response to a user requestto view underlying data related to the search result; a subsystem, atthe first device, implemented at least partially in hardware, that basedon receiving the at least one event identifier from the second device,sends at least one event record accessible by the first deviceassociated with the received at least one event identifier to the seconddevice, the at least one event record is comprised of raw data thatrelates to operations or activities in an information technologyenvironment.
 11. The apparatus as recited in claim 10, wherein eachevent record in the set of event records is associated with a timestamp.
 12. The apparatus as recited in claim 10, wherein each eventrecord in the set of event records is associated with a time stamp andis searchable based on a time represented by the time stamp.
 13. Theapparatus as recited in claim 10, wherein an event identifier includesan identification of both an event record and a device having access tothe event record.
 14. The apparatus as recited in claim 10, wherein thereceived at least one event identifier received by the first deviceincludes an identification of both an event record and the first device.15. The apparatus as recited in claim 10, wherein the search result sentby the first device to the second device does not include an eventrecord associated with an event identifier that was included in thesearch result.
 16. The apparatus as recited in claim 10, wherein thefirst device includes an indexer.
 17. The apparatus as recited in claim10, wherein the second device includes a search head.
 18. The apparatusas recited in claim 10, wherein the first device includes an indexer,and wherein the second device includes a search head.
 19. Anon-transitory computer-readable medium storing one or more sequences ofinstructions, wherein execution of the one or more sequences ofinstructions by one or more processors causes the one or more processorsto perform: receiving, at a first device, a search query to be performedon a set of event records accessible by the first device; searching, bythe first device, the set of event records using the search query;sending, by the first device, a search result to a second device, thesearch result including one or more event identifiers, wherein eachevent identifier of the one or more event identifiers is associated witha specific event record accessible by the first device of the set ofevent records that satisfied the search query, wherein each eventidentifier enables locating an associated specific event recordaccessible by the first device without searching the set of eventrecords; after sending the search result to the second device,receiving, by the first device from the second device, at least oneevent identifier that was included in the search result, the at leastone event identifier sent by the second device in response to a userrequest to view underlying data related to the search result; based onreceiving the at least one event identifier from the second device,sending, by the first device, at least one event record accessible bythe first device associated with the received at least one eventidentifier to the second device, the at least one event record iscomprised of raw data that relates to operations or activities in aninformation technology environment.
 20. The non-transitorycomputer-readable medium as recited in claim 19, wherein each eventrecord in the set of event records is associated with a time stamp. 21.The non-transitory computer-readable medium as recited in claim 19,wherein each event record in the set of event records is associated witha time stamp and is searchable based on a time represented by the timestamp.
 22. The non-transitory computer-readable medium as recited inclaim 19, wherein an event identifier includes an identification of bothan event record and a device having access to the event record.
 23. Thenon-transitory computer-readable medium as recited in claim 19, whereinthe received at least one event identifier received by the first deviceincludes an identification of both an event record and the first device.24. The non-transitory computer-readable medium as recited in claim 19,wherein the search result sent by the first device to the second devicedoes not include an event record associated with an event identifierthat was included in the search result.
 25. The non-transitorycomputer-readable medium as recited in claim 19, wherein the firstdevice includes an indexer.
 26. The non-transitory computer-readablemedium as recited in claim 19, wherein the second device includes asearch head.
 27. The non-transitory computer-readable medium as recitedin claim 19, wherein the first device includes an indexer, and whereinthe second device includes a search head.